To solve the performance bottlenecks such as time-consuming computation and inability of clustering in spectral clustering on large-scale datasets, a spectral clustering parallelization algorithm suitable for large-scale datasets was proposed based on Spark technology. Firstly, the similar matrices were constructed through one-way loop iteration to avoid double counting. Then, the construction and normalization of Laplacian matrices were optimized by position transformation and scalar multiplication replacement in order to reduce the storage requirements. Finally, the approximate eigenvector calculation was used to further reduce the computational cost. The experimental results on different test datasets show that, as the size of test dataset increases, the proposed algorithm has the running time of one-way loop iteration and the approximate eigenvector calculation increased linearly with slow speed, the clustering effects of approximate eigenvector calculation are similar to those of exact eigenvector calculation, and the algorithm shows good extensibility on large-scale datasets. On the basis of obtaining better spectral clustering performance, the improved algorithm increases operation efficiency, and effectively alleviates high computational cost and the problem of clustering.

In traditional pulmonary nodule detection algorithms, there are problems of low detection sensitivity and large number of false positives. To solve these problems, a pulmonary nodule detection algorithm based on deep Convolutional Neural Network (CNN) was proposed. Firstly, the traditional full convolution segmentation network was simplified on purpose. Then, in order to obtain high-quality candidate pulmonary nodules and ensure high sensitivity, the deep supervision of partial CNN layers was innovatively added and the improved weighted loss function was used. Thirdly, three-dimensional deep CNNs based on multi-scale contextual information were designed to enhance the feature extraction of images. Finally, the trained fusion classification model was used for candidate nodule classification to achieve the purpose of reducing false positive rate. The performance of algorithm was verified through comparison experiments on LUNA16 dataset. In the detection stage, when the number of candidate nodules detected by each CT (Computed Tomography) is 50.2, the sensitivity of this algorithm is 94.3%, which is 4.2 percentage points higher than that of traditional full convolution segmentation network. In the classification stage, the competition performance metric of this algorithm reaches 0.874. The experimental results show that the proposed algorithm can effectively improve the detection sensitivity and reduce the false positive rate.